Predicting Summary Quality using Limited Human Input

We present four experiments with summary evaluation approaches that use little or no human input in the form of model summaries or human judgements. We investigate whether system-produced summaries can be used to improve predictions of summary quality when few or no human summaries are available. We also validate our previous findings that measures of input-summary similarity and input cohesiveness are predictive of summary quality. We analyze the performance of our methods in predicting the human assigned scores for summarization systems from the 2008 and 2009 Text Analysis Conferences. Input-summary similarity metrics obtain correlations of about 0.7 with manual pyramid scores on the TAC ’09 data. Using only a collection of system summaries in place of gold standard, the correlation is 0.9. We also show that properties of input cohesiveness can predict the average system score with good accuracies.